1. Field of the Invention
The present invention relates to PCI Express system architectures and more specifically relates to modifications of the PCI Express architecture to enable interconnection of multiple independent PCI Express hierarchies to form a cluster of interconnected PCI Express hierarchies that may share access to endpoint devices in any of the clustered hierarchies.
2. Discussion of Related Art
The peripheral computer interconnect standards (“PCI”) have been popular interconnect standards for computing systems for many years. A more recent variant of the PCI architecture called “PCI Express” (“PCIe”) extends the family of PCI interconnect standards to include high speed serial exchange for bus transactions between master and slave devices within a system. In general, a system utilizing PCI interconnections (including PCIe) has a designated root device responsible for initially configuring parameters of the system and of the PCI interconnect between components of the system. In like manner, PCIe standards include such a root device that configures various aspects of the PCIe interconnect pathways as well as configuration parameters of the various endpoint devices within the system. The high speed serial nature of PCIe protocols and media allow for use of high speed switching devices as are generally known in the networking fields. Thus endpoint devices in a hierarchy with a root device may be more widely distributed and remain connected with other devices in the hierarchy through the PCIe switch/switches (i.e., a PCIe switched fabric connection). The root device in a PCIe hierarchy typically configures the switches of the fabric to allow switching of transactions based on various parameters of the applied transactions.
In parallel bus structure PCI systems as well as high speed serial PCIe systems, it is common that a system containing a single root device and any number of endpoint devices is referred to as a single PCI hierarchy. The root device of such a single hierarchy, as noted above, is generally responsible for configuring the interconnect media and protocols as well as various configuration attributes of the endpoint devices. Once so configured, various devices may temporarily assume the role of a master device in initiating a transaction with a slave device within the same hierarchy.
In older parallel bus structure PCI hierarchies, each PCI hierarchy is generally precluded from communicating with components of a second PCI hierarchy having another root device. Rather, each hierarchy is generally segregated and independent such that it has a single root device and its associated collection of endpoint devices. Such segregation of PCI hierarchies helps avoid problems with multiple root devices each attempting to initialize segments of the PCI bus structure and/or attempting to initialize the same endpoint devices with different configuration information. However, such segregation also generally precludes one hierarchy from easily accessing components within another hierarchy such that each hierarchy may share access to endpoint devices to thereby improve utilization of the processing and communication power therein. Though PCI bridge devices are known to allow coupling of otherwise segregated PCI bus segments, such devices add substantial cost and management complexity to a system.
Even with the switchable, serialized communications of the PCIe media and protocols, it is common to segregate PCI hierarchies such that one hierarchy cannot easily access endpoint devices in a second hierarchy. In fact, current PCIe standards and specifications preclude multiple root devices in a system and instead define a system as including only a single root device—i.e., a single hierarchy includes a single root device and its associated endpoint devices. Thus, multiple hierarchies are each operable completely independent of any other hierarchy and cannot share access to components in different hierarchies that each have a corresponding root device.
Although not presently standardized in a published, industry-adopted specification, recent developments by the PCISIG (PCI special interest group—the de facto industry standards organization defining PCI architectures) have begun to define an architecture that allows for multiple root devices in a PCIe system. Multi-root I/O structures have been proposed in the PCISIG such that multiple hierarchies may be physically integrated and physically interconnected within a single system—i.e., each hierarchy having a single specified root device and having associated endpoint devices all devices sharing common physical, switched connections. However, even these evolving standards still logically segregate the collection of endpoint devices such that each endpoint device may be accessed only by a single hierarchy within the system. In other words, each device is a member of a single hierarchy that includes a collection of devices associated only with that one hierarchy and its corresponding root device. Although multiple endpoint devices may share a single physical device and share a common interconnect they must be assigned to different hierarchies. Thus, although multiple hierarchies may be physically interconnected, no sharing of access is permitted between the various hierarchies of the system. Rather, the enhanced PCIe devices and switches logically segregate the devices—each to its own hierarchy—though the devices may share common physical communication paths through enhanced switching devices. In addition, for each hierarchy requiring access to a particular physical device there must be a separate logical endpoint device provide for each hierarchy. This requirement restricts the number of hierarchies that can access a physical device to the number of logical endpoint devices provided by the physical device.
Further, the evolving standards for multi-root virtualization and sharing evolving within the PCISIG involve substantial changes in PCI Express compliant switches and devices. The enhanced switches and devices are required for exchange of enhanced, modified PCIe packetized transactions. In other words, root devices and PCIe switches must be multi-root aware (“MRA” as defined in the evolving proposed standards) to allow the multiple hierarchies to coexist within a system sharing common physical attachments through enhanced PCIe switches. The evolving standards of PCISIG are documented at www.pcisig.com and are generally known to those of ordinary skill in the art of PCI system architectures.
To permit sharing of devices among a plurality of hierarchies, present solutions generally entail coupling the hierarchies of the system through other interconnection media and protocols to provide desired sharing of devices among the hierarchies. For example, Fibre Channel, Ethernet, and other well known network connectivity media and protocols may be employed to interconnect the various hierarchies to thereby provide shared access to endpoint devices in each hierarchy of the cluster. However, these and other networking solutions add substantial overhead to the processing for such shared access. Access to a remote device in another hierarchy must first be translated from a PCIe packetized transaction to another protocol for network exchange to the other hierarchy. Such protocol translations can add significant overhead processing to the desired shared access and thus may be practically unusable in high performance applications.
In view of the above discussion it clearly remains a problem to provide for sharing of endpoint devices among a plurality of PCIe hierarchies in a physical cluster each hierarchy having an associated root device.